{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——LightGBM\n",
    "原始特征+tfidf特征\n",
    "boosting_type参数取值goss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，LightGBM以进行参数调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/envs/tensorflow/lib/python3.7/site-packages/lightgbm/__init__.py:48: UserWarning: Starting from version 2.2.1, the library file in distribution wheels for macOS is built by the Apple Clang (Xcode_8.3.3) compiler.\n",
      "This means that in case of installing LightGBM from PyPI via the ``pip install lightgbm`` command, you don't need to install the gcc compiler anymore.\n",
      "Instead of that, you need to install the OpenMP library, which is required for running LightGBM on the system with the Apple Clang compiler.\n",
      "You can install the OpenMP library by the following command: ``brew install libomp``.\n",
      "  \"You can install the OpenMP library by the following command: ``brew install libomp``.\", UserWarning)\n"
     ]
    }
   ],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 读取数据\n",
    "# 这里使用原始特征+tf_idf特征，log(x+1)特征为原始特的单调变换，加上log特征对决策树模型影响不大\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "\n",
    "train1 = pd.read_csv(dpath +\"Otto_FE_train_org.csv\")\n",
    "train2 = pd.read_csv(dpath +\"Otto_FE_train_tfidf.csv\")\n",
    "\n",
    "#去掉多余的id\n",
    "train2 = train2.drop([\"id\",\"target\"], axis=1)\n",
    "train =  pd.concat([train1, train2], axis = 1, ignore_index=False)\n",
    "train.head()\n",
    "\n",
    "del train1\n",
    "del train2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字，LightGBM不支持字符串格式的特征输入/标签输入\n",
    "y_train = train['target'] #形式为Class_x\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#将类别的形式由Class_x变为0-8之间的整数\n",
    "\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM超参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM的主要的超参包括：\n",
    "1. 树的数目n_estimators 和 学习率 learning_rate\n",
    "2. 树的最大深度max_depth 和 树的最大叶子节点数目num_leaves（注意：XGBoost只有max_depth，LightGBM采用叶子优先的方式生成树，num_leaves很重要，设置成比 2^max_depth 小）\n",
    "3. 叶子结点的最小样本数:min_data_in_leaf(min_data, min_child_samples)\n",
    "4. 每棵树的列采样比例：feature_fraction/colsample_bytree\n",
    "5. 每棵树的行采样比例：bagging_fraction （需同时设置bagging_freq=1）/subsample\n",
    "6. 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)\n",
    "\n",
    "7. 两个非模型复杂度参数，但会影响模型速度和精度。可根据特征取值范围和样本数目修改这两个参数\n",
    "1）特征的最大bin数目max_bin：默认255；\n",
    "2）用来建立直方图的样本数目subsample_for_bin：默认200000。\n",
    "\n",
    "对n_estimators，用LightGBM内嵌的cv函数调优，因为同XGBoost一样，LightGBM学习的过程内嵌了cv，速度极快。\n",
    "其他参数用GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 相同的交叉验证分组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#直接调用lightgbm内嵌的交叉验证(cv)，可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证，且速度相对较慢\n",
    "def get_n_estimators(params , X_train , y_train , early_stopping_rounds=10):\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "     \n",
    "    lgbmtrain = lgbm.Dataset(X_train , y_train )\n",
    "     \n",
    "    #num_boost_round为弱分类器数目，下面的代码参数里因为已经设置了early_stopping_rounds\n",
    "    #即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=3,  metrics='multi_logloss' , early_stopping_rounds=early_stopping_rounds,seed=3 )\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:' , cv_result['multi_logloss-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 321\n",
      "best cv score: 0.48796097576768\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "n_estimators_1 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. num_leaves & max_depth=7\n",
    "num_leaves建议70-80，搜索区间50-80,值越大模型越复杂，越容易过拟合\n",
    "相应的扩大max_depth=7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed:  1.7min remaining:   49.8s\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed:  2.4min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(boosting_type='goss', class_weight=None,\n",
       "                                      colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=321,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=31,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=1.0,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'num_leaves': range(50, 90, 10)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10) #50,60,70,80\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4889353127485571\n",
      "{'num_leaves': 80}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'mean_train_score'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-9bcdb0775b77>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mtest_means\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'mean_test_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtest_stds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'std_test_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mtrain_means\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'mean_train_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mtrain_stds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'std_train_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'mean_train_score'"
     ]
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_means, yerr=test_stds,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_means, yerr=train_stds,label = ' Train')\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'Log Loss' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 性能抖动，取系统推荐值：70, 不必再细调"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. min_child_samples\n",
    "叶子节点的最小样本数目\n",
    "\n",
    "叶子节点数目：70，共9类，平均每类8个叶子节点\n",
    "每棵树的样本数目数目最少的类（稀有事件）的样本数目：200 * 2/3 * 0.7 = 100\n",
    "所以每个叶子节点约100/8 = 12个样本点\n",
    "\n",
    "搜索范围：10-50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 再往下细调"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  2.7min remaining:   40.4s\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:  3.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(boosting_type='goss', class_weight=None,\n",
       "                                      colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=321,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=1.0,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'min_child_samples': range(1, 10, 2)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(1,10,2) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4963814527570176\n",
      "{'min_child_samples': 9}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'mean_train_score'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-e33e26c4c3e6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mtest_means\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'mean_test_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtest_stds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'std_test_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mtrain_means\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'mean_train_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mtrain_stds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'std_train_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'mean_train_score'"
     ]
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### min_child_samples=10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 列采样参数 sub_feature/feature_fraction/colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed:  2.3min remaining:   34.0s\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed:  2.8min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(boosting_type='goss', class_weight=None,\n",
       "                                      colsample_bytree=1.0,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=10,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=321,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=1.0,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':10,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4916122380117515\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'mean_train_score'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-16-f3823637d2cb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mtest_means\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'mean_test_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtest_stds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'std_test_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mtrain_means\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'mean_train_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mtrain_stds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcv_results_\u001b[0m\u001b[0;34m[\u001b[0m \u001b[0;34m'std_train_score'\u001b[0m \u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'mean_train_score'"
     ]
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再调小一点，由于特征包括原始特征+tfidf特征，是多了些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 2 candidates, totalling 6 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   3 out of   6 | elapsed:   36.4s remaining:   36.4s\n",
      "[Parallel(n_jobs=4)]: Done   6 out of   6 | elapsed:   59.6s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score=nan,\n",
       "             estimator=LGBMClassifier(boosting_type='goss', class_weight=None,\n",
       "                                      colsample_bytree=1.0,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=10,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=321,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=70,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=1.0,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='deprecated', n_jobs=4,\n",
       "             param_grid={'colsample_bytree': [0.3, 0.4]},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':10,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(3,5)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.49166760500312634\n",
      "{'colsample_bytree': 0.4}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### colsample_bytree=0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)感觉不用调了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 减小学习率，调整n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 2678\n",
      "best cv score: 0.4749688837256359\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':10,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用所有训练数据，采用最佳参数重新训练模型\n",
    "由于样本数目增多，模型复杂度稍微扩大一点？\n",
    "num_leaves增多5\n",
    "#min_child_samples按样本比例增加到15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.4,\n",
       "               importance_type='split', learning_rate=0.01, max_bin=127,\n",
       "               max_depth=7, min_child_samples=15, min_child_weight=0.001,\n",
       "               min_split_gain=0.0, n_estimators=2678, n_jobs=4, num_class=9,\n",
       "               num_leaves=75, objective='multiclass', random_state=None,\n",
       "               reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "               subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'n_estimators':n_estimators_2,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':75,\n",
    "          'min_child_samples':15,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "lg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import _pickle as cPickle\n",
    "\n",
    "cPickle.dump(lg, open(\"Otto_LightGBM_goss_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"columns\":list(feat_names), \"importance\":list(lg.feature_importances_.T)})\n",
    "df = df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>feat_67_tfidf</td>\n",
       "      <td>25146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>feat_25_tfidf</td>\n",
       "      <td>23815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>feat_24_tfidf</td>\n",
       "      <td>22679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>feat_48_tfidf</td>\n",
       "      <td>22164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>feat_40_tfidf</td>\n",
       "      <td>21094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>feat_86_tfidf</td>\n",
       "      <td>20300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>feat_14_tfidf</td>\n",
       "      <td>17736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>feat_62_tfidf</td>\n",
       "      <td>16519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>feat_16_tfidf</td>\n",
       "      <td>14709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>feat_64_tfidf</td>\n",
       "      <td>14528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>feat_88_tfidf</td>\n",
       "      <td>14076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>feat_33_tfidf</td>\n",
       "      <td>13858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>feat_15_tfidf</td>\n",
       "      <td>13388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>feat_67</td>\n",
       "      <td>13220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>feat_42_tfidf</td>\n",
       "      <td>12693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>feat_70_tfidf</td>\n",
       "      <td>12559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>feat_54_tfidf</td>\n",
       "      <td>12352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>feat_34_tfidf</td>\n",
       "      <td>11506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>feat_8_tfidf</td>\n",
       "      <td>11079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>feat_24</td>\n",
       "      <td>10924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>feat_32_tfidf</td>\n",
       "      <td>10902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>feat_72_tfidf</td>\n",
       "      <td>10635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>feat_43_tfidf</td>\n",
       "      <td>10540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>feat_25</td>\n",
       "      <td>10501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>feat_60_tfidf</td>\n",
       "      <td>10065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>feat_48</td>\n",
       "      <td>9659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>feat_86</td>\n",
       "      <td>9450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>feat_85_tfidf</td>\n",
       "      <td>9426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>feat_36_tfidf</td>\n",
       "      <td>9285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>feat_71_tfidf</td>\n",
       "      <td>9177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>feat_30</td>\n",
       "      <td>1195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>feat_3</td>\n",
       "      <td>1177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>feat_84_tfidf</td>\n",
       "      <td>1174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>feat_7</td>\n",
       "      <td>1162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>feat_45</td>\n",
       "      <td>1087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>feat_23</td>\n",
       "      <td>1079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>feat_27</td>\n",
       "      <td>1032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>feat_49</td>\n",
       "      <td>1021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>feat_82_tfidf</td>\n",
       "      <td>1003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>feat_63</td>\n",
       "      <td>996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>feat_19</td>\n",
       "      <td>892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>feat_6_tfidf</td>\n",
       "      <td>864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>feat_21</td>\n",
       "      <td>832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>feat_52</td>\n",
       "      <td>816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>feat_46</td>\n",
       "      <td>788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>feat_61_tfidf</td>\n",
       "      <td>774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>feat_77</td>\n",
       "      <td>748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>feat_2</td>\n",
       "      <td>720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>feat_51_tfidf</td>\n",
       "      <td>708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>feat_12</td>\n",
       "      <td>665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>feat_31</td>\n",
       "      <td>658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>feat_5</td>\n",
       "      <td>657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>feat_81</td>\n",
       "      <td>614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>feat_93</td>\n",
       "      <td>593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>feat_28</td>\n",
       "      <td>574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>feat_61</td>\n",
       "      <td>377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>feat_84</td>\n",
       "      <td>370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>feat_82</td>\n",
       "      <td>245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>feat_6</td>\n",
       "      <td>223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>feat_51</td>\n",
       "      <td>199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           columns  importance\n",
       "159  feat_67_tfidf       25146\n",
       "117  feat_25_tfidf       23815\n",
       "116  feat_24_tfidf       22679\n",
       "140  feat_48_tfidf       22164\n",
       "132  feat_40_tfidf       21094\n",
       "178  feat_86_tfidf       20300\n",
       "106  feat_14_tfidf       17736\n",
       "154  feat_62_tfidf       16519\n",
       "108  feat_16_tfidf       14709\n",
       "156  feat_64_tfidf       14528\n",
       "180  feat_88_tfidf       14076\n",
       "125  feat_33_tfidf       13858\n",
       "107  feat_15_tfidf       13388\n",
       "66         feat_67       13220\n",
       "134  feat_42_tfidf       12693\n",
       "162  feat_70_tfidf       12559\n",
       "146  feat_54_tfidf       12352\n",
       "126  feat_34_tfidf       11506\n",
       "100   feat_8_tfidf       11079\n",
       "23         feat_24       10924\n",
       "124  feat_32_tfidf       10902\n",
       "164  feat_72_tfidf       10635\n",
       "135  feat_43_tfidf       10540\n",
       "24         feat_25       10501\n",
       "152  feat_60_tfidf       10065\n",
       "47         feat_48        9659\n",
       "85         feat_86        9450\n",
       "177  feat_85_tfidf        9426\n",
       "128  feat_36_tfidf        9285\n",
       "163  feat_71_tfidf        9177\n",
       "..             ...         ...\n",
       "29         feat_30        1195\n",
       "2           feat_3        1177\n",
       "176  feat_84_tfidf        1174\n",
       "6           feat_7        1162\n",
       "44         feat_45        1087\n",
       "22         feat_23        1079\n",
       "26         feat_27        1032\n",
       "48         feat_49        1021\n",
       "174  feat_82_tfidf        1003\n",
       "62         feat_63         996\n",
       "18         feat_19         892\n",
       "98    feat_6_tfidf         864\n",
       "20         feat_21         832\n",
       "51         feat_52         816\n",
       "45         feat_46         788\n",
       "153  feat_61_tfidf         774\n",
       "76         feat_77         748\n",
       "1           feat_2         720\n",
       "143  feat_51_tfidf         708\n",
       "11         feat_12         665\n",
       "30         feat_31         658\n",
       "4           feat_5         657\n",
       "80         feat_81         614\n",
       "92         feat_93         593\n",
       "27         feat_28         574\n",
       "60         feat_61         377\n",
       "83         feat_84         370\n",
       "81         feat_82         245\n",
       "5           feat_6         223\n",
       "50         feat_51         199\n",
       "\n",
       "[186 rows x 2 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFIpJREFUeJzt3X/sZXV95/Hna0FMt+oCMhAC2EEz\nbUqbLNIJkrht3LqBAXc72K0byEYmLpvpNrDRbDdxrMlitCa4G21CYmkwToTGimzVMFnGxQkha5oU\nZEDkR5HOFKmMzMLoUCVxo4t97x/3891e5tzvj/n+Oufe+3wkN/fe9z3nft/n3HPP654f935TVUiS\nNO4f9d2AJGl4DAdJUofhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOk7tu4HVOuuss2rr\n1q19tyFJU+Xhhx/+flVtWW64qQ2HrVu3cvDgwb7bkKSpkuRvVzKcu5UkSR2GgySpw3CQJHUYDpKk\nDsNBktRhOEiSOpYNhyQXJLk/yVNJnkzy/lb/SJLvJXm0Xa4aG+dDSQ4neTrJFWP1Ha12OMmesfqF\nSR5McijJF5Octt4TKklauZVsObwC/H5V/TJwGXBDkovaY39UVRe3y36A9tg1wK8AO4A/TnJKklOA\nTwNXAhcB1449zyfac20DXgKuX6fpkyStwrLhUFVHq+qRdvtl4CngvCVG2QncWVU/qarvAIeBS9vl\ncFU9U1U/Be4EdiYJ8JvAn7fxbweuXu0ESZLW7qSOOSTZCrwVeLCVbkzyWJK9Sc5otfOA58ZGO9Jq\ni9XfCPxdVb1yQl2SNtzWPfewdc89fbcxOCsOhySvA74EfKCqfgTcCrwFuBg4CnxyYdAJo9cq6pN6\n2J3kYJKDx44dW2nrkqSTtKJwSPIaRsHw+ar6MkBVvVBVP6uqvwc+w2i3EYw++V8wNvr5wPNL1L8P\nnJ7k1BPqHVV1W1Vtr6rtW7Ys+7tRkqRVWsnZSgE+CzxVVZ8aq587Nti7gSfa7X3ANUlem+RCYBvw\nDeAhYFs7M+k0Rget91VVAfcDv9PG3wXcvbbJkiStxUp+lfXtwHuBx5M82mp/wOhso4sZ7QJ6Fvhd\ngKp6MsldwF8xOtPphqr6GUCSG4F7gVOAvVX1ZHu+DwJ3JvlD4JuMwkiS1JNlw6Gq/oLJxwX2LzHO\nx4GPT6jvnzReVT3DP+yWkiT1zG9IS5I6DAdJUofhIEnqMBwkSR2GgySpw3CQ5ow/FaGVMBwkSR2G\ngySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcpDnhl990MgwHSVKH4SBJ6jAcJEkdhoOk\n3m3dc4/HRAbGcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQtK78MttsMBwk\nSR2GgySpw3CQJHUYDpKkjmXDIckFSe5P8lSSJ5O8v9XPTHIgyaF2fUarJ8ktSQ4neSzJJWPPtasN\nfyjJrrH6ryV5vI1zS5JsxMRKklZmJVsOrwC/X1W/DFwG3JDkImAPcF9VbQPua/cBrgS2tctu4FYY\nhQlwE/A24FLgpoVAacPsHhtvx9onTZK0WsuGQ1UdrapH2u2XgaeA84CdwO1tsNuBq9vtncAdNfIA\ncHqSc4ErgANVdbyqXgIOADvaY2+oqr+sqgLuGHsuSZppQz3196SOOSTZCrwVeBA4p6qOwihAgLPb\nYOcBz42NdqTVlqofmVCXJPVkxeGQ5HXAl4APVNWPlhp0Qq1WUZ/Uw+4kB5McPHbs2HItS5JWaUXh\nkOQ1jILh81X15VZ+oe0Sol2/2OpHgAvGRj8feH6Z+vkT6h1VdVtVba+q7Vu2bFlJ65KkVVjJ2UoB\nPgs8VVWfGntoH7BwxtEu4O6x+nXtrKXLgB+23U73ApcnOaMdiL4cuLc99nKSy9rfum7suSRJPTh1\nBcO8HXgv8HiSR1vtD4CbgbuSXA98F3hPe2w/cBVwGPgx8D6Aqjqe5GPAQ224j1bV8Xb794DPAT8H\nfLVdJEk9WTYcquovmHxcAOCdE4Yv4IZFnmsvsHdC/SDwq8v1IknaHH5DWpLUYThIkjoMB0lSh+Eg\nSeowHCRJHYaDJKnDcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR2GgySpw3CQpsRQ/xG9\nZpPhIEnqMBwkSR2GgySpw3CQJHUYDpKkDsNBktRhOEiSOgwHSVKH4SBJ6jAcJM00v1m+OoaDJKnD\ncJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR3LhkOSvUleTPLEWO0jSb6X5NF2uWrssQ8l\nOZzk6SRXjNV3tNrhJHvG6hcmeTDJoSRfTHLaek6gJOnkrWTL4XPAjgn1P6qqi9tlP0CSi4BrgF9p\n4/xxklOSnAJ8GrgSuAi4tg0L8In2XNuAl4Dr1zJBkqS1WzYcqurrwPEVPt9O4M6q+klVfQc4DFza\nLoer6pmq+ilwJ7AzSYDfBP68jX87cPVJToMkaZ2t5ZjDjUkea7udzmi184DnxoY50mqL1d8I/F1V\nvXJCfaIku5McTHLw2LFja2hdkrSU1YbDrcBbgIuBo8AnWz0Thq1V1CeqqtuqantVbd+yZcvJdSxJ\nWrFTVzNSVb2wcDvJZ4D/0e4eAS4YG/R84Pl2e1L9+8DpSU5tWw/jw0uSerKqLYck547dfTewcCbT\nPuCaJK9NciGwDfgG8BCwrZ2ZdBqjg9b7qqqA+4HfaePvAu5eTU/SPPFnqLXRlt1ySPIF4B3AWUmO\nADcB70hyMaNdQM8CvwtQVU8muQv4K+AV4Iaq+ll7nhuBe4FTgL1V9WT7Ex8E7kzyh8A3gc+u29RJ\n0piFUH325nf13MnwLRsOVXXthPKiK/Cq+jjw8Qn1/cD+CfVnGJ3NJEkaCL8hLUnqMBwkSR2GgyQN\nxJBONDAcJEkdhoOkZQ3pE602h+EgSeowHCRJHYaDJKljVb+tJKkf7vvXZnHLQYPnClHafIaDJKnD\ncJAkdRgOkqQOw0HShtm65x6PGU0pw0GSTmCoGQ6SpAkMB0lTY94/zW8mw0GS1GE4SJI6DAdJUofh\nIEnqMBwkSR2GgySpw3CQtCJ+MWy+GA6SpA7DQZLUYThIkjoMB6ln7sfXEBkOA+JKQkPjMjm/DAdJ\nUofhIEnLmMctKMNhYDyXXLPIZXr6LBsOSfYmeTHJE2O1M5McSHKoXZ/R6klyS5LDSR5LcsnYOLva\n8IeS7Bqr/1qSx9s4tyTJek+kJOnkrGTL4XPAjhNqe4D7qmobcF+7D3AlsK1ddgO3wihMgJuAtwGX\nAjctBEobZvfYeCf+LUnSJls2HKrq68DxE8o7gdvb7duBq8fqd9TIA8DpSc4FrgAOVNXxqnoJOADs\naI+9oar+sqoKuGPsuSRJPVntMYdzquooQLs+u9XPA54bG+5Iqy1VPzKhPlGS3UkOJjl47NixVbYu\nadZ5jGPt1vuA9KTjBbWK+kRVdVtVba+q7Vu2bFlli5Kk5aw2HF5ou4Ro1y+2+hHggrHhzgeeX6Z+\n/oS6JKlHqw2HfcDCGUe7gLvH6te1s5YuA37YdjvdC1ye5Ix2IPpy4N722MtJLmtnKV039lwaEDfT\npfly6nIDJPkC8A7grCRHGJ11dDNwV5Lrge8C72mD7weuAg4DPwbeB1BVx5N8DHioDffRqlo4yP17\njM6I+jngq+0iSerRsuFQVdcu8tA7JwxbwA2LPM9eYO+E+kHgV5frQ+rD1j338OzN7+q7Dc2goS9b\nfkNa0qq4q3G2GQ6StIh5DkDDYQ7N8wIvjfO9sDjDYZ34g3mSZonhIEnqMBwk6SRt9p6CPvZMGA7S\nAGzUm99dnVotw0GS1GE4SJI6DAdpSnmGnDaS4SBNOQNCG8FwkKSeDTHgl/3hPUnq2xBXnrPOLQdJ\nUofhIEnqMBykAXI3ivpmOKh3rgil4TEcJG2aIX8QGHJvfTAcNCi+QaeTr9vsMRwkrRtDYnYYDtIM\nWelPavjTG+tjlueh4SBJ6jAcJEkdhsM6m+XNTOlE7p6aXYbDBvANI2naGQ6SBscPV/0zHCRJHYbD\nlPMT1ubxNFHNE8NB0obYyIA0fDee/+xH0swwNNaPWw6SppqBsDEMhxWahgVwGnpcME299s15NTum\n6bVcUzgkeTbJ40keTXKw1c5MciDJoXZ9RqsnyS1JDid5LMklY8+zqw1/KMmutU2SJGmt1mPL4Z9X\n1cVVtb3d3wPcV1XbgPvafYArgW3tshu4FUZhAtwEvA24FLhpIVC08abpk4ykzbMRu5V2Are327cD\nV4/V76iRB4DTk5wLXAEcqKrjVfUScADYsQF9aQnrFRJ9hI2njmqWDGV5Xms4FPC1JA8n2d1q51TV\nUYB2fXarnwc8NzbukVZbrC4taghvnnk1r/N+KCvtzbLWU1nfXlXPJzkbOJDk20sMmwm1WqLefYJR\nAO0GeNOb3nSyvUqaIfO0ou7DmrYcqur5dv0i8BVGxwxeaLuLaNcvtsGPABeMjX4+8PwS9Ul/77aq\n2l5V27ds2bKW1qVBcoWnoVh1OCT5+SSvX7gNXA48AewDFs442gXc3W7vA65rZy1dBvyw7Xa6F7g8\nyRntQPTlraYVmrfNXalP8/JeW8tupXOAryRZeJ4/q6r/meQh4K4k1wPfBd7Tht8PXAUcBn4MvA+g\nqo4n+RjwUBvuo1V1fA19SVJvtu65h2dvflffbazZqsOhqp4B/umE+g+Ad06oF3DDIs+1F9i72l42\n0rx8SpDmyUaswGdtXeE3pCX1amgr1aH10xfDYUZ5HEIny+VlWPp+PQwHaRF9vzmlPhkOmklDW7EP\nrR9pOYaDJKnDcJCkDTDNv1cGhoMkaQLDQYPlGVfzx9d7OAwHaYNN2wpv2vrVxjAcVsFPtNPD12l+\n+FqvL8NB66KvN6YrBGljzGU4bNYKxRXX/FjNa+3ysbmc3ydnrf/sR2uwsLBO+gGw9f5hsKX+loZp\n2k+F1HSbyy0HTac+V3LTtoKdhn6noceVmqVpWWA4rIPlFoxZXHBWYl6nW0vzhI7pYDjMoNW+8eb9\nTTuP0z9v06uVMxx00tayQpn3ldG8T/9qTPM8m+YPHIbDJpvWBWWWzOprMKvTpX4YDjPOFYaWM82f\nbufNZr5Onso6pzZyIZvW55b0D9xyOEmunE7eZs+zjf57C8/vsqBZ5paDljRLK8D1/mLhwnNKs8gt\nB80MV9Trx+MQMhy0KeZxZTNv06vZYjhoLs1jWEknw3BopmFFMQ09rpd5mlZpiOb+gPRSK6EhrqD8\ndVVpug1xvTKJWw4DMLSAGlo/kjbf3G85bBZXqktz/miWbcRp1BvNcOjJNKwMV9LjNEzH0DjPNo/z\nevXmerfSiQvOyZ7B4j+fmQ6emTSdfM36NdfhMC18k2yc9Z63vlaaFYMJhyQ7kjyd5HCSPX33I0nz\nbBDhkOQU4NPAlcBFwLVJLuq3K220oe7uGWJP0mYbRDgAlwKHq+qZqvopcCews69mXDmszCzNp1ma\nFmk9DCUczgOeG7t/pNUkST1IVfXdA0neA1xRVf++3X8vcGlV/ccThtsN7G53fwl4eg1/9izg+2sY\nfzMMvceh9wf2uF7scX0MocdfqKotyw00lO85HAEuGLt/PvD8iQNV1W3AbevxB5McrKrt6/FcG2Xo\nPQ69P7DH9WKP62MaelwwlN1KDwHbklyY5DTgGmBfzz1J0twaxJZDVb2S5EbgXuAUYG9VPdlzW5I0\ntwYRDgBVtR/Yv4l/cl12T22wofc49P7AHteLPa6PaegRGMgBaUnSsAzlmIMkaUDmLhyG+DMdSS5I\ncn+Sp5I8meT9rf6RJN9L8mi7XNVzn88mebz1crDVzkxyIMmhdn1Gj/390ti8ejTJj5J8oO/5mGRv\nkheTPDFWmzjfMnJLWz4fS3JJjz3+tyTfbn18Jcnprb41yf8Zm59/0mOPi762ST7U5uPTSa7osccv\njvX3bJJHW72X+bhiVTU3F0YHu/8GeDNwGvAt4KIB9HUucEm7/Xrgrxn9jMhHgP/cd39jfT4LnHVC\n7b8Ce9rtPcAn+u5z7LX+38Av9D0fgd8ALgGeWG6+AVcBXwUCXAY82GOPlwOnttufGOtx6/hwPc/H\nia9te/98C3gtcGF735/SR48nPP5J4L/0OR9Xepm3LYdB/UzHgqo6WlWPtNsvA08xPd8Q3wnc3m7f\nDlzdYy/j3gn8TVX9bd+NVNXXgeMnlBebbzuBO2rkAeD0JOf20WNVfa2qXml3H2D0/aPeLDIfF7MT\nuLOqflJV3wEOM3r/b6ilekwS4N8AX9joPtbDvIXD4H+mI8lW4K3Ag610Y9us39vnLpumgK8lebh9\nWx3gnKo6CqOQA87urbtXu4ZXvwmHNB9h8fk21GX03zHaollwYZJvJvlfSX69r6aaSa/tEOfjrwMv\nVNWhsdqQ5uOrzFs4ZEJtMKdrJXkd8CXgA1X1I+BW4C3AxcBRRpukfXp7VV3C6Ndzb0jyGz33M1H7\nIuVvAf+9lYY2H5cyuGU0yYeBV4DPt9JR4E1V9VbgPwF/luQNPbW32Gs7uPkIXMurP7AMaT52zFs4\nrOhnOvqQ5DWMguHzVfVlgKp6oap+VlV/D3yGTdgsXkpVPd+uXwS+0vp5YWG3R7t+sb8O/78rgUeq\n6gUY3nxsFptvg1pGk+wC/iXwb6vtKG+7an7Qbj/MaH/+L/bR3xKv7dDm46nAbwNfXKgNaT5OMm/h\nMMif6Wj7Ij8LPFVVnxqrj+9rfjfwxInjbpYkP5/k9Qu3GR2sfILR/NvVBtsF3N1Ph6/yqk9oQ5qP\nYxabb/uA69pZS5cBP1zY/bTZkuwAPgj8VlX9eKy+JaP/wUKSNwPbgGd66nGx13YfcE2S1ya5kFGP\n39js/sb8C+DbVXVkoTCk+ThR30fEN/vC6GyQv2aU0h/uu5/W0z9jtMn7GPBou1wF/CnweKvvA87t\nscc3Mzr741vAkwvzDngjcB9wqF2f2fO8/MfAD4B/MlbrdT4yCqqjwP9l9In2+sXmG6PdIZ9uy+fj\nwPYeezzMaL/9wjL5J23Yf92WgW8BjwD/qsceF31tgQ+3+fg0cGVfPbb654D/cMKwvczHlV78hrQk\nqWPeditJklbAcJAkdRgOkqQOw0GS1GE4SJI6DAdJUofhIEnqMBwkSR3/D1qd8heE6ZwKAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x104f82d50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(lg.feature_importances_)), lg.feature_importances_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tfidf的特征重要性更高一些。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
